Spontaneous quantum Hall effect in quarter-doped Hubbard model on honeycomb 
lattice and its possible realization in quarter-doped graphene system 
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We show that as the result of the nesting property of the Fermi surface, the quarter-doped Hubbard 
model on honeycomb lattice is unstable with respect to the formation of a magnetic insulating state 
with nonzero spin chirality for infinitesimally small strength of electron correlation. The insulating 

2 

state is found to be topological nontrivial and to have a quantized Hall conductance of a^y — 
We find the Fermi surface nesting is robust for arbitrary value of next-nearest-neighbor hopping 
integral. It is thus very possible that the quarter-doped graphene system will realize such an exotic 
ground state. We also show that the quarter-doped Hubbard model on honeycomb lattice is in exact 
equivalence in the weak coupling limit with the 3/4-filled Hubbard model on triangular lattice, in 
which similar effect is also observed. 

PACS numbers: 75.10.-b, 73.43.-f, 71.27.+a 



Effects of electron correlation on honeycomb lattice 
have attracted a lot of interest recently [l|-|5[- The honey- 
comb lattice has the smallest coordination number of 3 
for a two dimensional lattice and has Dirac-type disper- 
sion at half filling. It is a nontrivial task to understand 
the role of electron correlation in such a background. As 
an example, it is recently reported that the transition 
from the semimetal phase to the antiferromagnetic or- 
dered phase in the half-filled Hubbard model on honey- 
comb lattice is realized in a two-step manner. An exotic 
spin liquid state is found in a small but finite correlation 
range in between the semimetal phase and the antiferro- 
magnetic ordered phase. 

However, in the two dimensional graphene sheet [IjQ, 
which is thought to be the most natural realization of 
an electron system with a honeycomb lattice, the corre- 
lation effect is believed to be rather weak. Such a view 
is in general reasonable. However, when the Fermi en- 
ergy approaches the Van Hove singularity (VHS) of the 
density of state, the importance of the correlation effect 
is much enhanced. Especially, when the system has a 
perfectly nested Fermi surface, an infinitesimally small 
interaction is enough to induce dramatic correlation ef- 
fect. Recently, large electron doping on graphene system 
has been achieved in experiment through both chemical 
doping and electrolytic gating 13, HI • It is also found by 
angle- resolved-photoemission measurement that the VHS 
is much more extended than the free electron prediction. 

The density of state of the free electron on honeycomb 
lattice (with nearest-neighbor hopping) is shown in Fig.l. 
The VHS appears on both sides of the half-filled back- 
ground(at 3/4 or 5/4 band filling^l^]). At such special 
fillings, the system has a perfectly nested Fermi surface. 
As an example, the single particle energy on the Brillouin 
zone boundary is given by ±t, where t is the hopping in- 
tegral between neighboring sites. It is important to note 
that rather than one, there are three independent nest- 
ing vectors: Qi = ^bi, Q2 = \h2 and Q3 = ^(bi + b2). 



where bi and b2 are the two elementary reciprocal vec- 
tors of the system(see Fig.l). The exact same situation 
also occurs on triangular lattice when the band is |- 
filled. In that case, an interesting magnetic insulating 
state with nonzero spin chirality and spontaneous quan- 
tum Hall effect is predicted (Til . In this paper, we find 
that the exact same thing will also happen on the hon- 
eycomb lattice. In fact, we can show that both systems 
share the exact same low energy theory near the VHS, 
apart from an overall reduction of the energy scale by a 
factor of 2. We also find that the nesting property of the 
Fermi surface is robust for arbitrary value of the next- 
nearest-neighbor hopping integral. We thus argue that 
the quarter-doped graphene system has large chance to 
realize such an exotic spontaneous quantum Hall state. 

In a magnetic ordered background with nonzero spin 
chirality, the motion of electron along a closed loop will 
experience a Berry phase(equals to the solid angle sub- 
tended by the spins along the loop), whose effect is in- 
distinguishable from the Aharonov-Bohm phase induced 
by external magnetic field. Such a quantum phase is pre- 
dicted to induce anomalous contribution to the Hall coef- 
ficient in systems with non-coplanar magnetic order [isl- 



15|. In particular, when the magnetic order opens a full 



gap in the electron spectrum, such anomalous contribu- 
tion can only take quantized values. 

For the |-filled Hubbard model on triangular lattice, as 
the result of the Fermi surface nesting, a four-sublattice 
non-coplanar magnetic order is established for infinites- 
imally small value of electron correlation. In this state, 
the ordered moments in the four sublattices point along 
the normals of the four surfaces of a tetrahedron. The 
spins on each elementary triangular plaquette subtend 
a solid angle of tt. In such a non-coplanar state, the 

electron spectrum opens a full gap and the Berry phase 

2 

effect induces a quantized Hall conductance of (J^y — ^ 
at zero temperature. Since the order is deduced from 
a weak coupling instability related to the nesting prop- 



FIG. 1: The density of state and the Fermi surface of free 
electron on honeycomb lattice with nearest-neighbor hopping. 
The VHS appears at E = ±f, which corresponds to electron 
filling of 3/4 and 5/4 per unit cell. 



FIG. 2: Illustration of the tetrahedron magnetic ordering pat- 
tern on honeycomb lattice. The directions of the ordered mo- 
ments have been chosen in such a way to make the mean field 
Hamiltonian as simple as possible. 



erty of the Fermi surface, it should be stable when the 
correlation is not too strong. 

Since the system has three independent nesting vectors 
at the VHS, it has many different choices for developing 
a magnetic order. The simplest choice of Bose condensa- 
tion at a single nesting vector results in a collinear mag- 
netic state, while the non-coplanar magnetic state with 
the tetrahedron ordering pattern can be viewed as a state 
with Bose condensation on all the three nesting vectors 
with the same strength [l6|. The tetrahedron ordering 
pattern can be perfectly fitted into the honeycomb lattice 
as shown in Fig. 2. In such a state, each site is neighbored 
by three sites in the other three magnetic sublattices. We 
note that the system can also be viewed as being com- 
posed of two triangular sublattices, on each of which a 
tetrahedron ordering pattern is established. 

To check the stability of the tetrahedron ordering pat- 
tern, we have carried out unrestricted mean field search 
for the quarter-doped Hubbard model on honeycomb lat- 
tice with 12 X 12 X 2 lattice sites. We have included both 
magnetic and charge order parameters in the optimiza- 
tion. On the 12 X 12 X 2 lattice, there are in total 864 
variational parameters for the spin density and 288 varia- 
tional parameters for the charge density to be optimized. 



We have used both the conjugate gradient method and 
the simulated annealing method to find the minimum 
of the variational energy. Both methods predict that 
the tetrahedron ordering pattern is the most stable one 
and the charge distribution is always uniform. We now 
present the mean field theory of this state. 

In the mean field treatment, the Hubbard model on 
honeycomb lattice has the form, 

H = -t i4.A.a(^J-B,a + h.C.) (1) 

<i,j>,cr 

— — < Ji,A > -Oi^A ^ 2^ < Jj,B > -^j.B, 

i 3 

here Si^A(Sj,_B) denotes the spin operator on site of 
sublattice A{B). 

The ordered moments in the tetrahedron state can be 
written as 

< S,^ ^(e.e^^-«- + e.e'Q-«' + e,e'Q-«-) (2) 

< s,,B >= ^(e,e'Q-«^ -e,e^Q-«^ -e,e^Q-i^O- 
v3 

With Eq.(2), the mean field Hamiltonian in the momen- 
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turn space takes the form 
H= J2 '/'?,k(rk-/3M)V^i,k+<^i,(rk-/3M)V'ii,k, (3) 
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The spectrum is 

thus exphcitly two- fold degenerate. Tk and M are 8x8 
Hermitian matrices and are given by 
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and /I is the chemical potential to be determined by 
filling concentration. In the summation, the momen- 
tum is restricted to the magnetic Brillouin zone given 
by — 5 < fci , ^2 < 5 (here we have used the convention 
for momentum in which k — kihi + fc2b2). 

The self-consistent equation for the order parameter m 
is given by 

^ ^ [(V'IkMVi.k) + (V-iVkWii.k)] . (4) 
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V 

^ keMBZ 



We have solved this equation as a function of [//t at zero 
temperature for the quarter-doped system. The result 
is shown in Fig. 3. The order parameter m is seen to 
increase with U /t from U/t ^ 0. This is the result of the 
nesting property of the Fermi surface. The tetrahedron 
order is thus a weak coupling instability of the system. 

The tetrahedron state has a nonzero spin chirality, i.e., 
{Si ■ {Sj X Sfc)) ^ for spins on neighboring sites. Electron 
moving on such a background will experience a nonzero 
Berry phase, which will induce an anomalous contribu- 
tion to the Hall response of the electron system. Since 
the state is fully gapped, such an anomalous contribution 
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must take quantized values of the form where n is 
the total Chern-number of all the occupied bands. Now 
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we show that a^y = ^ at zero temperature in this state. 

The Kubo formula for the Hall conductance a^y is 
given by 

e^2n , 



h iS (en(k) 

k.n.m 
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(/(e„(k))-/(e„,(k))), 
(5) 




FIG. 3: The order parameter as a function of U/t at zero 
temperature. The black line gives the result of the quarter- 
filled honeycomb lattice, the red line gives the result of the 
3/4-filled triangular lattice for a comparison. The inset shows 
the comparison in the weak coupling limit, in which the two 
lattices show the exact same behavior. 



in which S is the area of the system, „j (k) and ^ (k) 
are the matrix elements of the current operators between 
the n-th and the m-th eigenstate of the band Hamilto- 
nian at momentum k. /(e„i(k)) is the Fermi distribution 
function. The current operator j'^ is given by 
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V'JkCj^V'i.k + V'n.kCk'/'ii.k. 



(6) 



is given by a similar expression with replaced by 
C^. Here C^'^ are 8x8 matrices and are given by 



in which cj^'^ = 



x,y 
Ck 









^k+Qi 






c 






x.y 
k+Q2 









x.y 
^k+Qa 



-i2-Kk2 



y _ itV3i 



v^ 

^x.y 

k 



= it - ^(e" 



-12x^2 ~l 



To calculate axy , we first solve the mean field equation 
at finite temperature. The order parameter as a function 
of temperature at fixed doping and = 4 is shown in 
Fig. 4. The mean field critical temperature is found to be 
Tc w 0.09t for this set of parameters. The Hall conduc- 
tivity is also shown in Fig. 4. axy is found to be zero above 
Tc and begins to increase below Tc, following the exact 
same trend as the order parameter. It finally saturates 

2 

to the quantized value of axy = x ^'^^'^ temperature. 

The results obtained here are very similar to the results 
for the 3/4-filled Hubbard model on triangular lattice 
For example, the critical temperatures on both lattices 
are around 0.09^ for y = 4. To understand this close 
similarity, we note that the honeycomb lattice is actually 
composed of two interpenetrating triangular lattices, on 
each of which a tetrahedron ordering pattern is estab- 
lished. Actually, we can show that the quarter-doped 
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FIG. 4: The order parameter and the Hall conductance as 
functions of temperature at = 4. Note the close similarity 
of these results with the results on triangular lattice p^. 

Hubbard model on honeycomb lattice and 3/4-filled Hub- 
bard model on triangular lattice are described by the 
same theory in the low energy limit, except for a rescal- 
ing of the overall energy scale by a factor of 2. First, 
we note the dispersion of the honeycomb lattice has the 
form 

EI^ = t\g{k) \ = t^3 + 2(cos(A:i) + cos(fc2) + cos(fci - fca)), 

while the dispersion of the triangular lattice has the form 

El — — 2t(cos(fci) + cos(fc2) + cos(fci — fc2)). 

It can then be shown that both lattices share the same 
dispersion relation near the VHS (where E^ = —2t) apart 
from a factor of 2 rescaling of the energy scale. At the 
same time, it can be shown that the Hubbard interaction 
term on both lattices can be rescaled into each other in 
the low energy limit and the rescaling factor is also 2. 
In the low energy limit, the Hubbard interaction Hu = 
U 'T'i.t^j.i on triangular lattice is given by 

Hu r^U 4^,t^k^+Q-tcL,iCk.+Q.,;, (7) 

where the sum over i denotes the sum over the three 
patches of nested Fermi surfaces. ki,kj are the momen- 
tums on the i—th patch of nested Fermi surface and Q; 
denotes the nesting vector on this patch of Fermi surface. 
On honeycomb lattice, the Hubbard interaction takes the 
form oi Hu = U I]^(n^A,tf^^A,; + riiB ,-\niB ,\) ■ In the low 
energy limit, it reduces to 

+^ H 4,k.,t'^s,k;+Q,.t4,k;,iCi3,k.+Q.,i. (8) 

ki,k^,i 



The free electron system on honeycomb lattice forms two 
bands with eigen-energy i?±,k = ±^|ff(k)|. If we denote 
the operators of the two bands as c+ k and c_ ki we have 

in which (/)k is the phase of ^(k). Noting that the nested 
Fermi surface patches are given by fci = ^2 = \ 
and fci — fc2 = 5, it is straightforward to show that 
0ki — 0ki+Qi = TT, TT, and on the three patches. Thus, 
when the Hubbard interaction term is projected into the 
subspace of the E+ (or E- ) band, the phase of its matrix 
element will be exact zero. Collecting all contributions, 
we find that in the low energy limit the Hubbard inter- 
action takes the form 

Hu ^ 1^ Y c^,k.,t^+.M+Q"tc+,kj4C+_k,+Q,,i, 

ki:k;,i 

+ ^ H cL^k.,tC-,k;+Q.,tC-,k^4C_,k.+Q,4- (10) 
ki,k'.,i 

Thus in the low energy limit, the quarter-doped Hub- 
bard model on honeycomb lattice can be rescaled into 
the 3/4-filled Hubbard model on triangular lattice with 
a reduction of overall energy scale by a factor of 2. This 
explains the close similarity between the results obtained 
on both lattices. Note that this equivalence only holds 
in the weak coupling limit(see Fig. 3). 

Although the calculation is done at the mean field 
level, our conclusion that the ground state of the system 
is a topological insulator with a quantized Hall conduc- 
tance should be robust for sufficiently small ^■ , since the 
magnetic ordering is induced by the Fermi surface nest- 
ing. The situation at finite temperature is more subtle, 
since the Wagner- Mermin theorem prohibits spontaneous 
breaking of continuous symmetry at finite temperature in 
two dimensional systems. However, as the ordered state 
also breaks the discrete time reversal symmetry, possi- 
bility still exists that the spin chirality may survive even 
without ordered moment [18|. It is interesting to see if the 
spin chirality can survive at finite temperature, when the 
ordered moment is already washed out by thermal fluc- 
tuation. 

Finally, we turn to the implications of our results on 
the physics of graphene. First, we note that the disper- 
sion of the graphene system is not perfectly particle-hole 
symmetric. Uncertainty remains in the strength of longer 
ranger hopping integrals. The next-neighbor hopping in- 
tegral is estimated to be in the range of 0.02t < t' < 
O.2t0. A small but finite next-next-neighbor hopping in- 
tegral can also exist jl9j . However, we want to emphasis 
that the nesting property of the quarter-doped system is 
robust against the introduction of next-neighbor hopping 
integral. The perfect nesting will only be destroyed by 
the next-next-neighbor hopping, which is much smaller 
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than the nearest and next-nearest neighboring hopping 
integrals. We thus beheve that our result should be ap- 
plicable to the graphene system. 

To be more specific, we have done the mean field cal- 
culation for nonzero t' /t. The order parameter at zero 
temperature as a function of U/t for t' /t = 0.2 and 
t' /t = —0.2 are shown in Fig. 5, in which they are com- 
pared with the result for t'/t = 0. It is seen that the 
tetrahedron magnetic order still develops at infinitesi- 
mally small value of interaction, no matter what is the 
sign of t' . The sign of t' does play a role in reducing 
or enhancing the magnitude of the order parameter. On 
the honeycomb lattice, the sign of t' will change under 
the particle-hole transformation. For example, electron 
doping in a system with positive t' is equivalent to hole 
doping in a system with negative t' . Thus both signs of 
t' are meaningful for the graphene system. 




FIG. 5: The order parameter as a function of U/t for t'/t = 
0.2,0, and -0.2. 

After the completion of this work'2^, we noticed many 
related recent researches in which other ordering patterns 
are proposed 21 - 2^ . Especially, a chiral d-wave super- 



conducting order is suggested for the ground state of the 
quarter-doped system by RG analysis 23^ i25j . However, 
more recent functional RG study and variational calcu- 
lation show that for the quarter-doped system, the tetra- 
hedron magnetic ordering pattern proposed in this paper 
enjoys a much larger condensation energy than the chiral 
d-wave superconducting statep6|. 

We conclude that the quarter-doped Hubbard model 
on honeycomb lattice has weak coupling instability to 
the formation of a magnetic insulating state with nonzero 
spin chirality and quantized Hall conductance as a result 
of the nesting property of the Fermi surface. We find 
the nesting property is robust against the introduction of 
next-nearest-neighboring hopping terms. We thus believe 



that such an exotic state should be observable in quarter- 
doped graphene system. Such an exotic state can also act 
as the parent phase of even more exotic phases if we move 
slightly away from the VHS. 
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